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1. INTRODUCTION 

As 83% of the information a person receives from their environment comes through sight, vision is 
the most crucial component of human physiology [1]. According to World Health Organization (WHO) figures 
from 2011, there are 285 million people with visual impairment worldwide, of which 39 billionaire blind and 
246 have impaired vision. The walking cane, often known as a white cane or stick, and guide dogs are the most 
conventional and established mobility aids for those with vision impairments. The range of motion, the amount 
of information transmitted, and the need for training are these aids' main limitations. Modern technology is 
advancing quickly, and both the hardware and software fronts have the ability to offer capabilities for intelligent 
navigation. To assist the blind in navigating independently and safely, many electronic travel aids (ETA) have 
been developed recently. Additionally, cutting-edge technical options have lately been made available to assist 
blind people in independent navigation. The fact that ultrasound emitters and detectors are tiny enough to be 
carried without requiring a complicated circuit is another factor contributing to the technology's popularity [2]. 
The presence of too many impediments can be troublesome even for those without visual impairments [3], but 
it are especially bad for those who are blind. People with visual impairments frequently require outside help, 
which may come in the form of trained dogs, people, with specialised technological gadgets that act as decision- 
support systems. Existing sensors can detect and identify things that suddenly appear on the floor, but there is 
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also a significant risk from objects that are suddenly deep or from obstacles that are higher than waist level or 
stair cases [4]-[7]. The blind people find it exceedingly challenging to travel alone, and they run the danger of 
getting lost. As a result, there is no mechanism in place to locates a blind person, and using regular sticks won't 
allow the person to go around independently in public without things growing worse. This inspired us to create 
a stick-free solution for blind people utilising smart assistive glasses for the blind. These glasses will use 
computer vision, deep learning, and sensor fusion to let blind people navigate freely. Blind impaired person 
face the confront of always relying on someone for daily navigation [8]. 

In the proposed work, the Raspberry Pi single-board computer (SBC) is interfaced with the camera 
module which captures the image data from the surroundings. The sonar sensor is interfaced with the node 
MCU which detects the presence of obstacle in the proximity of the blind person [9]. The signal from the sensor 
is given to the Raspberry Pi which is then co related to the camera data to detect the obstacle. The camera then 
feeds the image to the trained neural network after preprocessing which will detect the type of obstacle in front 
of the blind person and return the results [10], [11]. The image data acts as an input data for determination of 
the type of obstacle in front of the blind person using sensor fusion when the obstacle is in the proximity [12], 
[13]. The board reading system is also implemented where the boards have a QR code which will be determined 
by the smart glasses and using the camera to scan the QR code and when the board is detected, the content on 
the board is read for the blind person using optical character recognition (OCR) and deep learning. 

The work is organized as: in section 2, literature review is discussed. Section 3 covers an overview of 
the work. Method is covered in section 4, section 5 presented with results and discussion, and at last in section 
6, we sum up the major points of the effort. 


2. LITERATURE REVIEW 

Ho et al. [14] advise an impediment detection technique that makes use of depth statistics to allow the 
visually impaired to keep away from boundaries after they pass in an unusual surroundings. The system is 
composed of three elements: scene detection, obstacle detection, and a vocal declaration. This take a look at 
proposes a new technique to cast off the floor plane that overcomes the over-segmentation hassle. This system 
addresses the over-segmentation trouble by doing away with the threshold and the initial seed function problem 
for the location growth technique using the connected component method (CCM). 

Lin et al. [15] suggested a navigation framework using smart phone application that can be used with 
an image recognition system. The said framework works in one of the two possible modes, online or offline, 
depending on network availability. When the system is turned on, the smartphone capture a picture and sends 
it to the server for processing. To differentiate between individual obstacles, the server uses deep learning 
algorithms. The main drawback of the system is high power consumption. Lock ef al. [16] investigated a 
multimodal user interface that uses sound andvibration alarms to transmit navigation information to target users. 
The main drawback is that you need to run arcore, which is not supported on all smartphone devices. 

Tanveer et al. [17] developed a walking aid for the visually impaired based on a special smartphone- 
enabled wearable device. Whenever thelocation of an obstacle is identified, the smartphone app plays a voice 
in Bengali/English language. GPS is explored to find the user's location, and the blind person's location is 
tracked using a Google Map. The overall error rate reported is approximately 5% for concrete and floor tiles. 
However, this method does not work under certain conditions. A strikingcase is a room with a raised floor. 

Chang et al. [18] presents a system for detecting air obstacles and road falls. In the event of a fall, 
emergency alert notifications are sent to family persons or defined guardians. The proposed system ccomprises: 
i) wearable smart glasses, ii) smart wand, iii) mobile app, and iv) cloud-based information management platform 
that sends relevant alerts. Experimental results claimed an average fall detection accuracy of up to 98.3%. 
However, the system cannot able to identify direct aerial and ground imagery such as road signs and traffic cones, 
and do not mention data about the power, cost, and weight of the proposed solution. 

Islam et al. [19] proposed the pedestrian guidance systems which can recognize obstacles in three directions 
such as left, front and right and potholes in the road surface using ultrasonic sensors in combination with 
convolutional neural network (CNN). It consists of: i) ultrasonic sensor, ii) Raspberry Pi, iii) Raspberry Pi camera, 
iv) headphones, and v) external power supply. The system is mounted on the user's head and receives feedback via 
sound signals. According to the authors, the system has accuracyof 98.73%, for the front sensor with an error rate of 
1.26% (obstacle distance 50 cm), while the image classification’s accuracy, precision and recall attained are 92.67%, 
92.33%, and 93%, respectively. However, the system requirements for headphones create problems for the blind and 
visually impaired. This is because headphones can potentially block out safety-threatening ambient sounds. 

Lin et al. [20] suggested a deep learning-based framework with an RGB-D camera, a semantic map, 
and an obstacle avoidance engine that learns from pilot input tasks. It consists of 1) smartphone, ii) head phones, 
iii) RGB-D stereo camera, iv) wearable terminal with sunglasses, and v) external PC. The system presents a 
voice interface to the user, weighs no more than 150 g, and achieves an accuracy of 98.7% in daytime, 97% in 
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daylight and 9% at night. The authors do not include any information about power requirements and cost. One 
of the system's weaknesses is its form factor, which affects sensitivity and fit to different lighting scenarios. 

Efficient and accurate scene text (EAST) stands for efficient and accurate scene text detector [21]. This 
method is a simple and robust pipeline that enables text detection in natural scenes and provides high accuracy 
and efficiency. Experimental results show that this method gives better results than previous methods in terms 
of accuracy and efficiency. 

Long et al. [22] present a fusion system for perception and obstacle avoidance. It consists of a 
millimeter wave radar and an RGB depth sensor, and also features a stereo user interface. Experiments with 
this system have shown that the effective detection range is increased to 80m compared to using only the RGB- 
D sensor. However, the proposed solutions are cumbersome and costly. Also, the system is limited to object 
detection, it doesn't recognize it, and it can't transmit because it's still running on the PC. ENVISION [23] uses 
a special approach to reliably and accurately detect static and dynamic obstacles in real-time video streams 
captured by smartphones with average hardware. The system can be further improved if the obstacle detection 
and classification module can help the target user better navigate the environment. 

Badrloo et al. [24] proposed a new approach to assist blind people with indoor and outdoor navigation 
by marking their location and guiding them to their destination. The system uses radio frequency identification 
(RFID) technology, which covers a distance of about 0.5 m, and the test results show that the accuracy of the 
proposed work is in the range of 1-2 m. However, the method(s) used to estimate the accuracy of the solution 
is not clearly defined. Meliones et al. [25] presented an obstacle detection process as a component of a mobile 
application that analyzes real-time data received from an external sonar. Its prime task is to discover obstacles 
in the user's path and transmit information about the detected distance, size and potential movement through a 
voice interface and advise the user on how to avoid the obstacles. 


3. THE PROPOSED METHOD 

The Smart assistive gadget for the blind people is depicted in Figure 1. The camera connected to the 
Raspberry Pi will continuously record the live video stream and transfer it to the neural network that has been 
taught. The blind person's path is obstructed by objects that are detected by the camera and close by objects 
that are detected by the sonar sensor using sensor fusion. The user's commands will be recognized as speech 
by the microphone, which will then direct the user via voice interaction based on the speech command received. 
The created speech recognition system will guarantee that a blind person may receive assistance more 
successfully by utilizing voice commands and will then give the blind person the essential information. 

The hardware architecture diagram shown in Figure 2 shows the smart assistive device for the blind. 
The Raspberry Pi-connected camera will continuously record the live video feed and send it to the trained 
neural network. The model was trained using transfer learning to forecast the class that the system is expected 
to recognize and infer. The camera checks the blind person's path for obstructions, and the sonar sensor uses 
sensor fusion to identify nearby objects. The blind individual will be alerted via audio feedback if the 
obstruction is too close. Additionally, using optical character recognition technology with openCV and 
application programming interfaces (API), the camera is in charge of detecting message boards and the posts 
made on them. The microphone will recognize the user's commands as speech, and based on the commands it 
has heard, it will then direct the user via voice interaction. By using voice instructions, the developed speech 
recognition system will ensure that a blind person may receive assistance more successfully and then provide 
the necessary information. 
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Figure 1. Overview of the work Figure 2. Hardware architecture 
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4. METHOD 

The algorithm of the proposed system is given in Algorithm 1. The solution consists of development 
of smart assistive glasses for blind using artificial intelligence. In the proposed work the Raspberry Pi SBC is 
interfaced with the camera module which captures the image data from the surroundings. The sonar sensor is 
interfaced with the Node MCU which detects the presence of obstacle in the proximity of the blind person. 


Algorithm 1. To detect the presence of obstacle and alert using deep learning 
als Initialize camera 
Initialize GPIO 
Initialize TTS(Text To Speech) 
Set Sensor as Input 
Initialize Microphone 
Read Data From Sensor and Detect Obstacle 
Check Mode 
. if ( Obstacle is Detected ) 
begin 
Capture Camera Frame 
Feed To Deep Learning Model 
Perform Inference 
Fetch Obstacle Type 
If (Obstacletype is known) 
Run Audio Feedback 
Alert “Obstacle Type” 
else 
Alert “Obstacle Message” 
End 
9. if ( mode =”OCR” ) 
begin 


arya OF WN 


Capture Camera Frame 
Correct End Distortions 
Perform OCR 
Play Detected text using TTS 
End 


In order to focus on improved development and administration, it is crucial to pick the ideal technique 
when creating a system. Traveling alone presents a significant challenge for those with visual impairments. 
Although efforts have been made to create intelligent helping sticks for the blind,carrying them around and 
their inherent limitations mean that the issues facing the visually impaired remain unresolved. This system 
suggests a more in genious solution to the afore mentioned issues. The creation of intelligent supportive glasses 
for blind persons is the main goal of this research. The suggested concept entails the creation of smart glasses 
that can help the blind navigate through daily activities. The resulting smart glasses have cameras that record 
video feeds of the environment and use deep learning and sensor fusion techniques to determine the types of 
obstacles in the blind person's path. 

Blind persons are alerted of barriers’ proximity through auditory alerts. Inorder to assist blind persons, 
the proposed design also uses OCR techniques to read the boards and messages on the boards. Thi project also 
includes voice assisted interaction using speech recognition, making it completely fail-proof and enabling voice- 
based interaction with smart glasses. As a result, this initiative uses sensor fusion and deep learning (DL) to give 
blind people a complete solution. There is no mechanism in place to notify blind individuals of the precise road 
conditions or the sort of impediment in front of them, and standard walking aids can not enable a blind person to 
move around independently in public without the situation getting worse. This inspires us to create a stick-free 
solution for blind people utilising smart assistance glasses for the blind. These glasses use computer vision, deep 
learning, and sensor fusion to help blind people navigate freely while also enabling optical character recognition. 


5. RESULTS AND DISCUSSION 

As a result, the Al-powered gadgets help people with impaired vision and blindness locate and identify 
various objects as well as read written text on a board. They benefit from more freedom, independence, and 
awareness of their environment. Voice commands can be used to aid the system. It will be a productive approach 
for blind individuals to use technology to engage with their surroundings and take advantage of its features. 


5.1. Testing of obstacle detection distance 


The different tests were carried out to detect the obstacle detection distance which is specified in the 
program as well as measure the accuracy. The table in Table 1 shows the actual v/s. the detected distance in cm while 
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the test is carried out using the ultrasonic sonar sensor. From the Table 1 and Figure 3, it can be concluded that the 
distance measurement using ultrasonic sensor is always accurate for greater distance with 98% accuracy. However 
for lower distance the accuracy drops tremendously. Considering the glasses will be used by blind person and the 
distance will never approach 0 immediately, the usage of this sensor is fine and is acceptable. 


Table 1. Result of test carried 
Trail# Actual distance (cm) __ Detected distance (cm) _ Accuaracy (in %) 


1 80 80 100 
2: 715 74 98.6 
3 70 70 100 
4 65 65 100 
3 60 59 98.6 
6 55 55 100 
7 50 50 100 
8 45 45 100 
9 40 40 100 
10 35 35 100 
11 30 30 100 
12 25 25 100 
13 20 19 78.6 
14 15 13 86 
15 10 8 80 
16 5 7 71 
17 3 0 NA 
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Figure 3. Results of test carried 


5.2. Detection of obstacles at different times of the day 

The detection of type of obstacle was carried out at different times of the day to check for the accuracy. 
This was done to understand the lighting conditions which affect the image processing tasks. Thus, from the graph 
Table 2, we can conclude that the detections were proper and 100% accurate only when we have better lighting 
conditions. In low lights the system designed will not work properly for type of obstacle detection. 


Table 2. Obstacle detection accuracy at different time 


Time Lighting conditon __ Detection result 
Early Morning Low-light 25% 
Morning Proper light 100% 
After noon Proper light 100% 
Evening Low light 38% 
Evening Artificial light 98% 


5.3. Text reading distance mapping for detection of the aurco tags on the boards 

The distance mapping was done for text recognition module to detect the optimum distance at which 
the blind glasses can detect and read text. The data collected is represented in the tabular format below. Thus, 
from the results as in Table 3, we can conclude that the detections are proper when the board to be read is at a 
distance of 1 to 2 meters from the glass. This can be increased by addition of 100x100 or 1,000x1,000 aurco 
tag on the board to be read. 
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Table 3. Text detection accuracy 
Trail no. Distance in mtrs _ Text detection 


1 5 No 
2 4 No 
3 3 No 
4 2 Yes 
5 1 Yes 
6 0.5 No 


6. CONCLUSION 

The proposed system is related to the concept of smart assistive glasses for the blind and visually 
impaired persons using artificial intelligence. From on the design, we can conclude that the proposed system can 
help the blind by providing them with smart glasses that enable them for day-to-day’s activities by acting as a 
third eye. Obstacle types and obstacle sensors can ensure that blind people are always aware of their environment 
and can navigate on their own safely. The implemented OCR system helps blind people read the text on the board 
and translate it into voice. As such, the proposed system surely serves as an influential tool to help blind people 
navigate freely. The only drawback of deep learning model is the lengthy inference process. To reduce the time 
to detect tangible obstacles, future systems may be built using deep learning hardware accelerators rather than 
Raspberry Pi. Additionally, a 4-layer PCB can be used to make the system more compact and reliable. 
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